Learning Bidirectional Action-Language Translation with Limited Supervision and Testing with Incongruent Input

Human infant learning happens during exploration of the environment, by interaction with objects, and listening to repeating utterances casually, which is analogous unsupervised learning. Only occasionally, a would receive matching verbal description an action it committing, similar supervised Such mechanism can be mimicked deep We model this weakly paradigm using our Paired Gated Autoencoders (PGAE) model, combines language autoencoder. After observing performance drop when reducing proportion training, we introduce Transformed (PTAE) Transformer-based crossmodal attention. PTAE achieves significantly higher accuracy in language-to-action action-to-language translations, particularly realistic but difficult cases only few training samples are available. also test whether trained behaves realistically conflicting multimodal input. In accordance concept incongruence psychology, conflict deteriorates output. Conflicting input has more severe impact than input, features lead larger interference. on mostly unlabeled data where labeled scarce, plausibly tested incongruent